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Career-Readiness Features in Korean Assessment 
Items! 


Kilchan Choi, Jenny C. Kao, Nichole M. Rivera, and Li Cai 


CRESST/University of California, Los Angeles 


Abstract: This report is the third in a series considering career-readiness 
features within high school assessments. The goal of this study was to 
explore international comparisons by applying feature analysis to Korean 
assessment items. Twenty math test items from the Gyeonggi Province in 
South Korea along with performance data from roughly 4,000 Grade 12 
examinees were obtained. We applied the process of feature rating and 
analyses explained in the first two reports in this series to the Korean 
assessment items. Math test items were translated and rated using the 
same scheme as the American items by the same math expert raters. 
While the mathematics content differed from the American test, career- 
readiness features were also found in this small set of items. Korean 
math items shared similar patterns of feature representation with 
Smarter Balanced items, with the exception of a couple of features (e.g., 
critical thinking and visualization were rated more frequently among 
Korean items). Critical thinking was positively associated with item 
difficulty. It is possible that the type of math content (e.g., calculus) plays 
a role, which should be explored in future studies with larger sets of test 
items, different math content, and other international data sets. 


Introduction 


The purpose of this study was to continue CRESST’s career-readiness feature analysis 
work by exploring international comparisons with assessment data from The Republic of Korea 
(hereafter, Korea). Korea is widely recognized for achievement in primary and secondary 
education, particularly in international assessments (So & Kang, 2014). For example, Korea 
regularly outpaces the United States and the international average on assessments such as PISA 
(Programme for International Student Assessment) in reading, science, and mathematics 
(Organisation for Economic Co-operation and Development [OECD], 2015). Since this 
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benchmarking study will focus on assessment metrics, some context about the educational 
policies and practices in Korea may be useful. 


The educational system in Korea has some similarities with education in the United States. 
The Korean education system follows a “6—3—3-4” model: six years of primary schooling, three 
years of middle school, three years of high school, and four years of undergraduate education. 
The Korean government offers both vocational and academic high school tracks, and started 
offering preschool for children in 2012 (National Center on Education and the Economy [NCEE], 
2018). Primary and secondary school are compulsory until the age of 15 (Korean Council for 
University Education, 2015; Kwon, Lee, & Shin, 2015). The compulsory education system in the 
U.S. roughly mirrors this pattern but can vary from state to state. Attendance in the U.S. is 
generally required from about age 6 until at least age 16, with students generally continuing 
education until they complete Grade 12 (National Center for Education Statistics, n.d.-a). 


As the U.S. struggles with educational reforms and the balance between learning, 
assessment, and career readiness, so too does Korea. Korean schools follow a national 
curriculum framework developed by the Ministry of Education (n.d.-a), which was revised in 
2015. The Korean government also passed the Career Education Act in 2015, to clarify the 
responsibilities in providing effective career education for students. The newer curriculum is 
aimed at reducing content and increasing choice in schools (So & Kang, 2014). Approximately 
20% of students enrolled in upper secondary schools are on vocational tracks, down from 40% 
in the late 1990s (NCEE, 2018). The curriculum which previously diverged into vocational and 
academic tracks has been merged into a single ladder pathway. To increase career awareness, 
Korean students are now required to spend at least one hour per week in career exploration 
beginning in lower secondary school (NCEE, 2018) and have “free learning semesters” aimed at 
career exploration beginning in middle school (International Centre for Career Development 
and Public Policy, 2017). 


Beyond secondary (high school) education, both the United States and Korea have higher 
than average numbers of 25- to 64-year-olds with tertiary (or postsecondary) education 
compared to other OECD (Organisation for Economic Co-operation and Development) member 
countries (46% and 47%, respectively; OECD, 2018). However, academic achievement may 
come at a cost. As of 2010, a reported 74% of Korean students engaged in private tutoring such 
as hagwons, or “cram schools,” leading to government interventions to reduce student stress 
(Ripley, 2011). Compared to youth in 30 other OECD countries, Korean youth spend an average 
of three more hours per day studying (“Korean youth study longest hours in OECD,” 2009). They 
sleep one less hour and exercise 22 minutes less than youth in five other OECD countries, 
including the United States. Beyond high school, the Korean Ministry of Education reports that 
about 71% of students go on to pursue tertiary education (Ministry of Education, n.d.-b). The 
emphasis on academic performance has led to unintended consequences such as highly 
stressed students, lower student interest in learning, increasingly deskilled teachers because of 


the prescribed national curriculum (So & Kang, 2014), and “excessive” college enrollment for 
available jobs (International Centre for Career Development and Public Policy, 2017). 


Similar to the U.S., Korea has a system of assessments that are administered nationally, by 
province (district in the U.S.) or at the school level. In California, students in Grades 3-8 and 11 
take the California Assessment of Student Performance and Progress (CASPP) including math 
and ELA (National Center for Education Statistics, n.d.-b). Korea has a system of assessments 
called the National Assessment of Educational Achievement (NAEA). Each year, the NAEA is 
administered to all students in Grades 9 and 11, in the subjects of Korean, math, and English. 
Tests in science and social studies are administered to a sample of students in Grade 9. Results 
are reported in aggregate and used to provide additional support for schools and inform policy 
(NCEE, 2018). 


The purpose of the present study was to provide a preliminary international benchmark, 
specifically, comparing the career-readiness features found in the Gyeonggi Province 
Mathematics Test with the career-readiness features found in the Grade 11 Smarter Balanced 
mathematics test administered in the state of California, which were reported in a companion 
report (Kao, Choi, Rivera, Madni, & Cai, 2018). The Korean test items used in this study were 
translated from a 12" grade mathematics test developed in alignment with national standards 
in order to evaluate educational accountability in the Gyeonggi Province (Gyeonggi Institute of 
Education, 2014). We applied the process of feature rating and analyses developed in our prior 
report to the Korean test items. As discussed in Kao et al. (2018), career-readiness features 
were found within high school assessment items, which suggests that inferences can be drawn 
about students’ career readiness from their test scores. We were interested in whether career- 
readiness features were also found in Korean items, and if so, which features. We were also 
interested in which particular features had relationships with item difficulty. Results could pave 
the pathway for future international benchmarking studies. 


Method 


Test Items 


For this study, we examined a 12th-grade-level mathematics achievement test from 
Korea, which was administered in 2014 by the Gyeonggi Institute of Education. The Gyeonggi 
Institute of Education is a Korean-government-funded educational research institute. This test 
consists of 20 multiple-choice items covering calculus concepts, including limits of a function, 
continuous functions, differentiation, higher order derivatives, application of differentiation 
and derivatives, indefinite integrals, definite integrals, and applications of indefinite and 
definite integrals. In addition, according to the test blueprint, test task domains include 
concepts and procedures, problem solving, communication, and inferences (Gyeonggi Institute 
of Education, 2014). 


The items were independently translated into English by three native Korean-speaking 
bilingual graduate students, with experience in both the Korean and United States educational 
systems. These translators were familiar with the math content and did not require initial 
content training. The translated items were then translated from English back into Korean 
(“back-translated”), as described by Brislin (1970). The translations were validated by a fourth 
native Korean speaker as free from errors due to literal translations or “meaning errors” 
(Brislin, 1970, p. 197). Translations were slightly edited for grammar and style. See the 
Appendix for the translated items. 


Target Career-Readiness Features 


Features for math test items. As described in our prior report on feature analysis, 
following subject-matter expert review, an initial list of 36 career-readiness features was 
created, and then narrowed by math content area experts into the final list of 22 features (Kao 
et al., 2018). Table 1 shows the list of 22 career-readiness features used to rate math items, 
with descriptions. They are grouped broadly into three categories: features related to skills, 
features related to abilities, and features related to either work activities or work context. 


Table 1 


Career-Readiness Target Features Used to Rate Math Test Items by Category 


Feature 


Description 


Features related to skills 


Active learning 


Complex problem solving 


Critical thinking 


Reading comprehension 


Understanding the implications of new information for both current 
and future problem solving and decision making. 


Identifying complex problems and reviewing related information to 
develop and evaluate options and implement solutions. 


Using logic and reasoning to identify the strengths and weaknesses of 
alternative solutions, conclusions or approaches to problems. 


Understanding written sentences and paragraphs in work-related 
documents. 


Features related to abilities 


Deductive reasoning 


Flexibility of closure 


Inductive reasoning 


Information ordering 


Mathematical reasoning 


Memorization 


Number facility 


Time sharing 


Written expression 


Visualization 


The ability to apply general rules to specific problems to produce 
answers that make sense. 


The ability to identify or detect a known pattern, figure, object, word, 
or sound that is hidden in other distracting material. 


The ability to combine pieces of information to form general rules or 
conclusions. 


The ability to arrange things or actions in a certain order or pattern 
according to a specific rule or set of rules. 


The ability to choose the right mathematical methods or formulas to 
solve a problem. 


The ability to remember information such as words, numbers, 
pictures, and procedures. 


The ability to add, subtract, multiply, or divide quickly and correctly. 


The ability to shift back and forth between two or more activities or 
sources of information. 


The ability to communicate information and ideas in writing so others 
will understand. 


The ability to imagine how something will look after it is moved 
around or when its parts are moved or rearranged. 


Features related to work activities/context 


Analyzing data and 
information 


Estimating the quantifiable 
characteristics of products, 
events, or information 


Identifying the underlying principles, reasons, or facts of information 
by breaking down information or data into separate parts. 


Estimating sizes, distances, and quantities; or determining time, 
costs, resources, or materials needed to perform a work activity. 


Feature Description 


Getting information Observing, receiving, and otherwise obtaining information from all 
relevant sources. 


Identifying objects, actions, Identifying information by categorizing, estimating, recognizing 
and events differences or similarities, and detecting changes in circumstances or 
events. 


Importance of being exact or Being very exact or highly accurate is important to performing this 
accurate job. 


Making decisions and solving Analyzing information and evaluating results to choose the best 


problems solution and solve problems. 

Organizing, planning, and Developing specific goals and plans to prioritize, organize, and 
prioritizing work accomplish your work. 

Processing information Compiling, coding, categorizing, calculating, tabulating, auditing, or 


verifying information or data. 


Feature rating. The three raters who previously rated Smarter Balanced math items, 
described in Kao et al. (2018), used the same procedure and feature list to rate the Korean test 
items. As described in the prior report, the math raters held doctorates or master’s degrees in 
fields related to engineering, measurement, and statistics and were familiar with the content. 
Test items were rated by all three raters using an ordinal scale of 1 to 4, with a rating of 1 
generally referring to little or no presence of the feature (in order to solve the problem), and 4 
generally referring to the feature being present or necessary to solve the problem. The 
Krippendorff’s alpha for interrater reliability ratings was .86 (Krippendorff, 2011). Raters 
discussed disagreements to achieve final consensus. As in the analyses on the Smarter Balanced 
items, ratings were subsequently recoded from ordinal values into binary values as the focus 
was on the frequency or presence of a feature. Ratings of 1 and 2 were recoded as 0, indicating 
the feature was not present, and ratings of 3 and 4 were recoded as 1, indicating the feature 
was present. The binary coding was used with item difficulty analyses. 


Results 


Descriptive Results 


We present descriptive results from the Grade 11 Smarter Balanced feature ratings 
alongside results from the Korean item feature ratings. Table 2 shows the means and standard 
deviations of career-readiness features using the original 4-point rating scale for Smarter 
Balanced and Korean math items. The range of standard deviations were similar for each set of 
test items. Based on descriptive results, Korean math items were about as high as Smarter 
Balanced math items on features such as reading comprehension, deductive reasoning, 
mathematical reasoning, number facility, analyzing data or information, getting information, 


importance of being exact or accurate, and processing information. They were also similarly low 
in features such as active learning, flexibility of closure, information ordering, memorization, 
time sharing, and written expression. 


Table 2 


Means and Standard Deviations of Career-Readiness Features Using Original 4-Point Rating 


Scale, Smarter Balanced vs. Korean Math Items 


Smarter Balanced Korean 
(121 items) (20 items) 
Feature M SD M SD 
Features related to skills 
Reading comprehension 3.1 0.7 2.8 0.8 
Complex problem solving 2.1 0.7 2.0 0.0 
Critical thinking 2.0 0.8 2.5 1.2 
Active learning 1.2 0.5 1.3 0.9 
Features related to abilities 
Deductive reasoning 3.6 0.6 3.0 0.0 
Number facility 2.8 1.2 3.6 0.9 
Mathematical reasoning 2.5 0.9 2.4 0.9 
Inductive reasoning 2.2 0.8 1.1 0.2 
Memorization 1.8 0.4 1.1 0.2 
Time sharing 1.7 0.7 1.4 0.6 
Flexibility of closure 1.4 0.6 1.2 0.4 
Visualization 1.3 0.7 1.6 0.9 
Information ordering 1.1 0.3 1.0 0.0 
Written expression 1.1 0.4 1.0 0.0 
Features related to work activities/context 
Analyzing data or information 3.1 0.7 3.4 0.5 
Processing information 2.6 0.5 3.0 0.0 
Identifying objects, action, and events 2.6 0.6 2.1 0.3 
Importance of being exact or accurate 2.5 0.8 3.0 0.8 
Organizing, planning, and prioritizing work 2.3 0.8 2.6 0.5 
Getting information 2.2 0.8 2.5 0.8 
Making decisions and solving problems 2.0 0.6 2.0 0.0 
Estimating the quantifiable characteristics of 2.0 0.7 1.1 0.4 


products, events, or information 


Note. Features are sorted in descending order of Smarter Balanced means within each category. 


Table 3 shows the frequency and percentage of each career-readiness feature using the 
binary ratings for both Smarter Balanced and Korean math items. Under the binary coding, the 
20 Korean math items contained between five and 12 career-readiness features, with a mean of 
7.95. In comparison, the 121 Smarter Balanced math items contained between two and 15 
features, with a mean of 7.77. Some features were rated more frequently among Smarter 
Balanced items (such as complex problem solving and inductive reasoning), while some 
features were rated more frequently among Korean items (such as critical thinking and 
visualization). 


Table 3 


Frequencies of Career-Readiness Features Using Recoded Binary Ratings (Smarter Balanced vs. 


Korean Math Items) 


Smarter Balanced Korean 
(121 items) (20 items) 
Feature Frequency % Frequency % 
Features related to skills 
Reading comprehension 97 80.2 12 60.0 
Complex problem solving 37 30.6 0 0.0 
Critical thinking 17 14.0 10 50.0 
Active learning 8 6.6 2 10.0 
Features related to abilities 
Deductive reasoning 116 95.9 20 100.0 
Number facility 81 66.9 18 90.0 
Mathematical reasoning 67 55.4 11 55.0 
Inductive reasoning 43 35.5 0 0.0 
Visualization 15 12.4 6 30.0 
Time sharing 10 8.3 1 5.0 
Flexibility of closure 6 5.0 0 0.0 
Information ordering 2 1.7 0 0.0 
Written expression 2 1.7 0 0.0 
Memorization 0 0.0 0 0.0 
Features related to work activities/context 
Analyzing data or information 98 81.0 20 100.0 
Importance of being exact or accurate 72 59.5 18 90.0 
Processing information 72 59.5 20 100.0 
Identifying objects, action, and events 62 51.2 2 10.0 
Organizing, planning, and prioritizing work 48 39.7 12 60.0 
Getting information 41 33.9 6 30.0 
Estimating the quantifiable characteristics of 25 20.7 1 5.0 
products, events, or information 
Making decisions and solving problems 21 17.4 0 0.0 
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Table 4 shows the total number of features rated as present in Korean math items by 
feature category. 


Table 4 
Frequency and Percentage of Features Rated as Present in Korean Math Items by Feature Category 


Features related to Features related to Features related to 
skills abilities work activities/context 
Number of 
features Frequency % Frequency % Frequency % 
0 5 25.0 - - - - 
1 8 40.0 1 5.0 - - 
2 7 35.0 6 30.0 - - 
3 - - 9 45.0 8 40.0 
4 - - 4 20.0 6 30.0 
5 - - - - 5 25.0 
6 - - - - 1 5.0 
Total 20 100.0 20 100.0 20 100.0 


Table 5 shows the career-readiness features rated by test item number. Each item 
contained between five and 12 features, with a total of 15 career-readiness features 
represented across the test items. 
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Table 5 


Career-Readiness Features by Test Item 


Test item number 


Feature 1 2 3 4 5 6 7 8 9 10 11 12 #13 #14 #15 #16 #17 «+18 «19 20 = Total 


Deductive reasoning xX xX X 20 


Analyzing data or information X xX X 20 


<x xX 
<x x«~ x«~ x 


Processing information X X X 20 
Number facility X X X X xX xX xX X 18 


< «—~ «Kx xK x 
< «—~ x«K~ KK x 
x< 
<x KK KX K xX 
<x «—_ «KK Ki xX 
x< 
x< 
<i «— KK Ki xX 
x< 
<x KK K KX 
x< 
x< 
<x K K KX 
x< 


x< 


Importance of being exact or xX xX 18 


accurate 


Reading comprehension X KX XK xX xX xX X XK XK xX xX xX 12 


Organizing, planning, and X xX xX xX X Mh XK % XK X xX xX 12 
prioritizing work 


Mathematical reasoning X ¥%h® xX KX X XK X X X xX X 11 


Critical thinking xX xX xX xX X Mh XK XK X X 10 


Visualization X X X X X X 


6 

Getting information X X X X xX X 6 | 

Active learning —— —_ YX a oe — Xx ss 
—_ : 


Identifying objects, actions, X X 
and events 


Time sharing X 1 


Estimating the quantifiable X 1 
characteristics of products, 
events, or information 
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Test item number 


Feature 1 2 3 4 5 6 7 


8 


13 


14 


15 


16 


17 


18 


19 20. Total 


Complex problem solving 
Flexibility of closure 
Inductive reasoning 
Information ordering 
Memorization 

Written expression 


Making decisions and solving 
problems 


oo 90 0'0,.0:0 


Total 5 8 10 7 12 10 6 
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Table 6 shows a correlation matrix of the feature rating for Korean math items. The matrix 
shows the extent to which a feature rated as present was correlated with other features rated 
as present. For example, if critical thinking was rated as present in a Korean item, such an item 
was likely to also be rated for visualization as well as organizing, planning, and prioritizing work. 
Note that features without any variability were not included. That is, deductive reasoning, 
analyzing data or information, and processing information were rated as present across all 20 
items and were excluded. Similarly, complex problem solving, flexibility of closure, inductive 
reasoning, information ordering, memorization, written expression, and making decisions and 
solving problems were not rated as present in any of the items and were also excluded. 


Unlike Smarter Balanced math items, some features showed strong correlations. For 
instance, the importance of being exact or accurate showed a negative 1.0 correlation with 
identifying objects, actions, or events. However, because the correlations are based on only 20 
items with binary values (present or not present), the correlations should be interpreted with 
some caution. 
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Table 6 


Correlation Matrix of Features Rated as Present in Korean Math Items 


Feature 1 2 3 4 5 6 7 8 9 10 11 
1. Active learning — 
2. Critical thinking .00 — 
3. Reading comprehension 27 .20 — 
4. Mathematical reasoning -.03 .10 -.12 — 
5. Number facility 11 33 .07 .03 — 
6. Time sharing .69** 23 19 21 .08 _ 
7. Visualization -.22 44 -.13 -.07 22 -.15 = 
8. Estimating -.08 23 19 21 .08 -.05 35 = 
9. Getting information 51* .22 54* = -.07 -.15 35 -.19 35 = 
10. Identifying, objects, actions, events -11 .00 27 -.03 -.44* -.08 15 .69** .51* = 
11. Importance of being exact or accurate  .11 .00 -.27 .03 .44* .08 -.15 -.69**  -.51* -1.00** = 
2 SOnganteins Planning ane: BMOnuZINE?: . 35 61** 38 -33 07 19 31 19 54* 927-27 


work 


*o < .05, two-tailed. **p < .01, two-tailed. 
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Relationship Between Career-Readiness Features and Item Difficulty 


One goal of the present study was to explore whether the career-readiness features 
within a test item was associated with the item’s difficulty. Item response theory (IRT) allows 
estimates of item difficulty and test-taker ability and is widely used in large-scale educational 
assessment (Thissen & Steinberg, 2009). We first examined the item difficulty parameter and 
discrimination parameters using the two-parameter item response model (2PL). Data were 
obtained from the 2014 Gyeonggi Institute of Education test administration, and included 3,949 
records examinee records. 


Item responses were coded as 0 for incorrect or 1 for correct. Descriptive statistics of the 
item difficulty parameters were computed. The mean item difficulty was 0.49 (SD = 0.69), with 
a minimum of -0.78 and a maximum of 1.65. Table 7 shows the distribution of item difficulty in 
the Korean items. As shown in the figure, more than three quarters of the items have positive 
values of item difficulty. However, as the minimum and maximum values of the item difficulty 
indicate, there were no items that were extremely easy or extremely difficult. 
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Table 7 


Item Difficulty in Korean Items 


Item Item difficulty (IRT-B) SE 


1 -0.78 0.04 
2 0.16 0.04 
3 0.35 0.03 
4 -0.45 0.03 
5 0.54 0.04 
6 1.43 0.08 
7 -0.32 0.03 
8 -0.26 0.03 
9 1.49 0.14 
10 0.75 0.05 
11 0.87 0.07 
12 0.72 0.06 
13 1.30 0.09 
14 0.89 0.05 
15 0.18 -0.03 
16 1.65 0.10 
17 -0.02 0.03 
18 0.71 0.05 
19 0.46 0.04 
20 0.40 0.03 


A multiple regression technique, imposing precision weights, was employed to examine 
the relationship between item difficulty and identified career readiness features. Weight was 
calculated by 1/([Standard Error of item difficulty]?/the total number of examinees). We first 
explored the relationship using a longer list of career-readiness features as predictors and item 
difficulty as the outcome. However, with a limited number of items and correlations among 
predictors, we then chose a reduced feature list—critical thinking, reading comprehension, 
math reasoning, and visualization. The weighted multiple regression result is presented in Table 
8. 


The critical thinking feature had a statistically significant positive relationship with item 
difficulty. In other words, the item difficulty increased by 0.53 if an item rated with the critical 
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thinking feature was compared to an item without it. In addition, an item with the visualization 
feature is considered more difficult by approximately 0.50 scale in difficulty than an item 
without the visualization feature. The R-square of this model was .56, which is considered 
substantial given the small number of predictors. 


Table 8 


Relationships Between Item Difficulty Estimate and Career-Readiness Features: 
Results From Weighted Multiple Regression for Korean Math Items 


Variable Estimate SE t p 
Intercept -0.034 0.195 -0.17 .866 
Critical thinking* 0.525 0.220 2.38 .031 
Visualization 0.497 0.233 2.14 .050 
Reading comprehension 0.205 0.195 1.05 .310 
Math reasoning -0.294 0.198 -1.49 .158 
Note. R square = 56. Features are sorted by coefficients, from positive to negative. 


*0 <.05. 


To ensure the appropriateness of model selection, we also employed a weighted multiple 
regression with stepwise selection method. As shown in Table 9, the results are similar to those 
in the previous model. The R-square of this model was .53. The result suggests that for 12th 
grade Korean students, items measuring features such as critical thinking or visualization are 
more difficult than items without such features. Items measuring features such as math 
reasoning tend to be easier than items without math reasoning present. 


Table 9 

Relationships Between Item Difficulty Estimate and Career Readiness Features: 
Results From Weighted Multiple Regression With Stepwise Selection Method 
for Korean Math Items 


Variable Estimate SE t p 
Intercept 0.095 0.153 0.38 544 
Critical thinking* 0.616 0.203 9.24 .008 
Visualization 0.451 0.229 3.88 .067 
Math reasoning -0.357 0.189 3.56 .078 
Note. R square = .53. Features are sorted by coefficients, from positive to negative. 


*0<.05. 


Results differ from those found in the 11th grade Smarter Balanced math items, in which 
several features contained coefficient estimates larger than .5 (positive: written expression, 
time sharing, math reasoning; negative: making decisions and solving problems). Korean math 
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items were all rated as O for written expression and for making decisions and solving problems, 
and only one of the 20 Korean items was rated as having time sharing. It is possible that Korean 
math items do not contain such features, though other test items from Korea may have yielded 
different results. 


Discussion 


In this report, we explore preliminary international comparisons with a 20-item multiple- 
choice math test from the Gyeonggi province in Korea. Similar to Smarter Balanced summative 
math test items (Kao et al., 2018), certain career-readiness features can also be found in Korean 
math test items. While the mathematics content differs from the American test, career- 
readiness features were also found in this small set of items. These features were present even 
though response types were multiple choice for Korean items and varied for Smarter Balanced 
items. This study represents a preliminary comparison of Korean and Smarter Balanced test 
items on mathematics and evidence that an international assessment can be rated using the 
feature rating process. 


However, some limitations of the study and differences in our samples require caution in 
making direct comparisons. For example, in terms of the features in the respective tests, a 
larger proportion of Korean items contained critical thinking and visualization, which were also 
positively associated with item difficulty. This may be reflective of differences in culture or 
educational pedagogy. However, this may also reflect differences in content. That is, while 
Smarter Balanced math item content was more focused on algebraic concepts, the Korean 
items generally covered calculus. Perhaps calculus requires more critical thinking and 
visualization than algebra. In informal conversations with the math raters, they remarked that 
the Korean items generally had more items that involved multiple steps to solve the problem 
(i.e., critical thinking) and that the Korean items were more likely to require either visualizing or 
drawing out graphs. Another limitation to this study is that the Korean test items were 
administered for local accountability and may not reflect the same attributes as summative 
tests used to gain entry at the university level. Apart from content differences, future research 
might consider individual differences such as age or motivation. 


Overall, this study represents a preliminary comparison of Korean and Smarter Balanced 
test items on mathematics and evidence that international assessment can be rated using the 
feature rating process. In light of educational focuses on career readiness in the U.S. and 
abroad, future research should explore a larger set of test items to explore whether trends 
continue, and whether there are implications of differing math content on career readiness for 
high school students. This may include calculus-based assessments and other mathematics 
content from other countries. Such research can shed additional light on drawing career- 
readiness inferences from mathematics assessments. 
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Appendix: 
Translated Test Items 


5x2-9 


1. Evaluate lim 
x— 00 


oe 


2x2 -3 


vx2+a-2 


x-1 


2. For two constants a and b, if lim 
x1 


® 3 
@4 
@5 
@6 
©7 


= Db, finda + 2b? 


3. The graphs of the functions y = f(x) and y = g(x) are shown in the figures below. Which of the 
following statements must be true? 


A. lim 22 =0 
x22+0 f(x) 


B. Jim, Fg) =0 
C. limt f(x) — g(x)} = 2 


OA 
@B 
@A,C 
@B,C 
© A,B,C 


22 


4. A function f (x) is continuous in the set of all real numbers. If f(x) satisfies (x — 2) f(x) = x? —5x + 
6), find f (2). 

© -3 

@ -1 

@2 

@4 

© 6 


5. The graphs of two functions y = f(x) and y = g(x), defined in the interval [—2, 2], are shown 
below. Which of the following statements must be true? 


A. A function f (x) g(x) is continuous at x = —1. 
B. A function (1 — x)g(x) is continuous at x = 1. 
C. A function f(x) + g(x) is continuous at x = 0. 


OA 
@c 
@A,B 
@B,C 
© A,B,C 


6. A function f(x) is continuous in the interval [1,2] and satisfies f(1) = 2a? — a and f(2) =a—3. 
What is the sum of all possible values of a which makes f(x) — x = 0 have at least one real root on the 
interval [1, 2]? 


7. For a differentiable function f(x) with f’(a) = 5, what is the value of lim A 


O6 
@9 
@ 12 
@15 
© 18 
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8. For function f (x) = (x? + 1)(2x — 1), find f'(1). 
®2 
@ 3 
@4 
@5 
@6 


9. For function f(x) = 3x* + 4ax? — 6x? + 12ax, for x > 1, what is the range of constant a? 
Oa>-1 

@a<-1 

@Oa<l 

®@a>1 

@®@-1<a<l 


10. A cubic function f(x) = x3 + ax? + bx +c has the following conditions: 


A. f(—x) = -f(@) 


B. At a point on y = f(x), the slope of the tangent line at x = 2 is 9. 


What is the local maximum of f(x)? 
O1 
@2 
@3 
@4 
©5 


11. The tangent line to the curve y = x? — 2 at point (0,0) meets the curve at a point of tangency A and 
point B. What is the length of AB? 

© V10 

@ 2V10 

@ 3V10 

@ 4V10 

© 5V10 


12. When a cubic function f(x) = 4x3 + 3x? is translated parallel to the Y-axis by k units and meets the 
X-axis at three different points, what is the range of k? 


O-1<k<3 
@-2<k <0 
Ok<-1k>-= 
@l<k<1 
Ok <-=,k>0 
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13. As shown in the figure below, let a point on curve y = —x? + 3x (0 < x < 3) be Pand let H be the 
point from point P to the X-axis, forming a perpendicular line. Point A is the intersection where the 
curve intersects with the x-axis other than the origin 0. Suppose the area of APAH is maximum. What is 
the length of a segment AH? 


5 
@1 
3 
Se 
@®2 
5 


14. Suppose a point P moves from the origin through a vertical line. Let x be the location of the point P 
at time t as shown in the figure, which depicts x = f(t). Which of the following statements below 
must be true? (Note that 0 < t < d) 


A. The point P shifts its direction twice while it moves through the line. 
B. The velocity of point P when it passes the origin the first time after departure is f’(b). 
C. The direction of the movement is identical when 0 <t <aand0<t<d. 


OA 
@B 
@A,C 
@B,C 
© A,B,C 
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15. A polynomial f (x) satisfies the given conditions: 
f'(x) = 6x? — 4x +1 and f(0) = 3 

What is the value of f(1)? 

O1 

@2 

@ 3 

@4 

©5 


16. For a function f(x) = x”, the sequence of points which divide the closed interval [0, 1] (including 
the endpoints) into n segments are 0 = Xo, X4,%2,°'',Xn—1,Xn = 1 in ascending order. Evaluate 

lim a earned (Note that n is positive integer equal to or greater than 2.) 

n—-0o 

1 


@® © 80 
ROlUWI NN] Rw 


2 
17. Evaluate the definite integral fo ax - Io ae. 


18. Suppose that lim f(t? + at + 3)dt = 11. What is the value of a constant a? 
x1 X- 


D6 
@7 
@8 
@9 
© 10 
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19. For all real numbers x, a polynomial f (x) is f(x) + f(—x) = 0 and satisfies the following 
conditions. What is the value of Vf @)dx? 


A. i f(x)dx = -2 
B. f/,f@ddx = -5 
6. ef@aras 


20. What is the area between a curve y = x* — 1 anda straight line y = x — 1? 


© ® ©8089 


27 


UCLA 


ec | 


NATIONAL CENTER FOR RESEARCH ON EVALUATION, 
STANDARDS, AND STUDENT TESTING 


National Center for Research on Evaluation, 
Standards, and Student Testing (CRESST) 


Graduate School of Education & Information Studies 
University of California, Los Angeles 
300 Charles E. Young Drive North 
GSE&IS Bldg., Box 951522 
Los Angeles, CA 90095-1522 


(310) 206-1532 
www.cresst.org 


